Image-based guidance for navigating tubular networks

ABSTRACT

Systems and methods for image-based guidance for facilitating navigation of tubular networks. A region of interest in three-dimensional image data may first be segmented. An endoscopic instrument may be detected in two-dimensional intraoperative image data of the region of interest. A centerline of the detected endoscopic instrument may then be determined. The endoscopic instrument and the centerline may be backprojected to generate a three-dimensional backprojected volume. A device path of the endoscopic instrument may be generated based at least in part on the three-dimensional backprojected volume and the three-dimensional image data.

TECHNICAL FIELD

The present disclosure generally relates to image data processing, andmore particularly to image-based guidance for navigating tubularnetworks.

BACKGROUND

Lung cancer is a frequent and deadly disease. Early, precise, fast andcost-effective diagnosis and treatment of lung lesions are important toenhance survival rates for patients. To perform diagnosis and therapy,clinicians typically use bronchoscopy to visualize the inside ofairways. Bronchoscopy involves the use of a bronchoscope to examine theairways of a patient for abnormalities (e.g., bleeding, tumors, lesions,foreign bodies, inflammation). The bronchoscope is usually inserted intoa patient's airway through the patient's nose or mouth and can extendinto the lungs of the patient.

Bronchoscopes, however, are limited in how far they may be advancedthrough the airway due to their size before becoming wedged in theairway of the patient. Where the bronchoscope is too large to reach atarget location deep in the lungs (e.g., beyond third-fourth generationof branching of airways), a clinician may utilize certain real-timeimaging modalities to initially determine the location of a targettissue and to confirm the location of the target tissue. If theclinician is unable to reach the target tissue for any reason, theclinician may insert a navigated percutaneous catheter to the confirmedlocation of the target tissue.

Traditional navigation techniques, however, have downfalls, such asoutdated imaging data, deformation of the airway that is not accountedfor and difficulty of use even for advanced users. Reliable and accurateguidance is important for success in biopsy and treatment procedures inbronchoscopy.

SUMMARY

Described herein are systems and methods for image-based guidance forfacilitating navigation of tubular networks. A region of interest inthree-dimensional image data may first be segmented. An endoscopicinstrument may be detected in two-dimensional intraoperative image dataof the region of interest. A centerline of the detected endoscopicinstrument may then be determined. The endoscopic instrument and thecenterline may be backprojected to generate a three-dimensionalbackprojected volume. A device path of the endoscopic instrument may begenerated based at least in part on the three-dimensional backprojectedvolume and the three-dimensional image data.

BRIEF DESCRIPTION OF THE DRAWINGS

A more complete appreciation of the present disclosure and many of theattendant aspects thereof will be readily obtained as the same becomesbetter understood by reference to the following detailed descriptionwhen considered in connection with the accompanying drawings.

FIG. 1 is a block diagram illustrating an exemplary system;

FIG. 2a shows an exemplary method of image-based guidance by a computersystem;

FIG. 2b shows an exemplary method of determining a device path;

FIG. 2c shows another exemplary method of determining a device path;

FIG. 3 shows various exemplary views reconstructed from athree-dimensional (3D) computed tomographic (CT) image volume of a lung;

FIG. 4 shows an exemplary two-dimensional (2D) fluoroscopic image of apatient's lungs;

FIG. 5 illustrates an exemplary backprojection of the detectedbronchoscope and its corresponding centerline;

FIG. 6 shows exemplary candidate segments; and

FIG. 7 shows an exemplary visualization.

DETAILED DESCRIPTION

In the following description, numerous specific details are set forthsuch as examples of specific components, devices, methods, etc., inorder to provide a thorough understanding of implementations of thepresent framework. It will be apparent, however, to one skilled in theart that these specific details need not be employed to practiceimplementations of the present framework. In other instances, well-knownmaterials or methods have not been described in detail in order to avoidunnecessarily obscuring implementations of the present framework. Whilethe present framework is susceptible to various modifications andalternative forms, specific embodiments thereof are shown by way ofexample in the drawings and will herein be described in detail. Itshould be understood, however, that there is no intent to limit theinvention to the particular forms disclosed, but on the contrary, theintention is to cover all modifications, equivalents, and alternativesfalling within the spirit and scope of the invention. Furthermore, forease of understanding, certain method steps are delineated as separatesteps; however, these separately delineated steps should not beconstrued as necessarily order dependent in their performance.

The term “x-ray image” as used herein may mean a visible x-ray image(e.g., displayed on a video screen) or a digital representation of anx-ray image (e.g., a file corresponding to the pixel output of an x-raydetector). The term “in-treatment x-ray image” as used herein may referto images captured at any point in time during a treatment deliveryphase of an interventional or therapeutic procedure, which may includetimes when the radiation source is either on or off. From time to time,for convenience of description, CT imaging data (e.g., cone-beam CTimaging data) may be used herein as an exemplary imaging modality. Itwill be appreciated, however, that data from any type of imagingmodality including but not limited to x-ray radiographs, MRI, PET(positron emission tomography), PET-CT, SPECT, SPECT-CT, MR-PET, 3Dultrasound images or the like may also be used in variousimplementations.

Unless stated otherwise as apparent from the following discussion, itwill be appreciated that terms such as “segmenting,” “generating,”“registering,” “determining,” “aligning,” “positioning,” “processing,”“computing,” “selecting,” “estimating,” “detecting,” “tracking” or thelike may refer to the actions and processes of a computer system, orsimilar electronic computing device, that manipulates and transformsdata represented as physical (e.g., electronic) quantities within thecomputer system's registers and memories into other data similarlyrepresented as physical quantities within the computer system memoriesor registers or other such information storage, transmission or displaydevices. Embodiments of the methods described herein may be implementedusing computer software. If written in a programming language conformingto a recognized standard, sequences of instructions designed toimplement the methods can be compiled for execution on a variety ofhardware platforms and for interface to a variety of operating systems.In addition, implementations of the present framework are not describedwith reference to any particular programming language. It will beappreciated that a variety of programming languages may be used.

As used herein, the term “image” refers to multi-dimensional datacomposed of discrete image elements (e.g., pixels for 2D images, voxelsfor 3D images, dynamic voxels or doxels for 4D datasets). The image maybe, for example, a medical image of a subject collected by computertomography, magnetic resonance imaging, ultrasound, or any other medicalimaging system known to one of skill in the art. The image may also beprovided from non-medical contexts, such as, for example, remote sensingsystems, electron microscopy, etc. Although an image can be thought ofas a function from R³ to R, or a mapping to R³, the present methods arenot limited to such images, and can be applied to images of anydimension, e.g., a 2D picture, 3D volume or 4D dataset. For a 2- or3-Dimensional image, the domain of the image is typically a 2- or3-Dimensional rectangular array, wherein each pixel or voxel can beaddressed with reference to a set of 2 or 3 mutually orthogonal axes.The terms “digital” and “digitized” as used herein will refer to imagesor volumes, as appropriate, in a digital or digitized format acquiredvia a digital acquisition system or via conversion from an analog image.

The terms “pixels” for picture elements, conventionally used withrespect to 2D imaging and image display, “voxels” for volume imageelements, often used with respect to 3D imaging, and “doxels” for 4Ddatasets can be used interchangeably. It should be noted that the 3Dvolume image is itself synthesized from image data obtained as pixels ona 2D sensor array and displays as a 2D image from some angle of view.Thus, 2D image processing and image analysis techniques can be appliedto the 3D volume image data. In the description that follows, techniquesdescribed as operating upon doxels may alternately be described asoperating upon the 3D voxel data that is stored and represented in theform of 2D pixel data for display. In the same way, techniques thatoperate upon voxel data can also be described as operating upon pixels.In the following description, the variable x is used to indicate asubject image element at a particular spatial location or, alternatelyconsidered, a subject pixel. The terms “subject pixel”, “subject voxel”and “subject doxel” are used to indicate a particular image element asit is operated upon using techniques described herein.

One aspect of the present framework facilitates guidance through atubular network via the use of image data. A tubular network generallyrefers to a luminal branched structure, such as a bronchial or lungnetwork, in which an endoscopic instrument (e.g., bronchoscope) may beinserted. The image data is acquired using an imaging technique, such asfluoroscopy. A guidance solution based on information obtained from theimage data alone advantageously enhances the workflow and positioning offixed or mobile C-arms for image acquisition.

Another aspect of the present framework facilitates guidance throughtubular networks via the use of image data and electromagnetic (EM)guidance. Traditional EM tracking techniques do not considerfluoroscopic or other image-based information to accurately depict theposition of the endoscopic instrument. Combining both informationsources advantageously yields improved accuracy. These and otherexemplary features and advantages will be described in more detailsherein.

It is understood that while a particular application directed totransbronchial network navigation may be shown, the technology is notlimited to the specific implementations illustrated. For example, thetechnology may be applied to other types of tubular networks, such asthe gastrointestinal tract, the ear, the urinary tract and so forth.

FIG. 1 is a block diagram illustrating an exemplary system 100. Thesystem 100 includes a computer system 101 for implementing the frameworkas described herein. Computer system 101 may be a desktop personalcomputer, a portable laptop computer, another portable device, amini-computer, a mainframe computer, a server, a cloud infrastructure, astorage system, a dedicated digital appliance, a communication device,or another device having a storage sub-system configured to store acollection of digital data items. In some implementations, computersystem 101 operates as a standalone device. In other implementations,computer system 101 may be connected (e.g., using a network) to othermachines, such as imaging device 102, workstation 103, endoscopicinstrument 130 and optional electromagnetic (EM) tracking system 135. Ina networked deployment, computer system 101 may operate in the capacityof a server (e.g., thin-client server, such as syngo®.via by SiemensHealthcare), a client user machine in server-client user networkenvironment, or as a peer machine in a peer-to-peer (or distributed)network environment.

Computer system 101 may include a processor device or central processingunit (CPU) 104 coupled to one or more non-transitory computer-readablemedia 105 (e.g., computer storage or memory), display device 108 (e.g.,monitor) and various input devices 110 (e.g., mouse or keyboard) via aninput-output interface 121. Computer system 101 may further includesupport circuits such as a cache, a power supply, clock circuits and acommunications bus. Various other peripheral devices, such as additionaldata storage devices and printing devices, may also be connected to thecomputer system 101.

The present technology may be implemented in various forms of hardware,software, firmware, special purpose processors, or a combinationthereof, either as part of the microinstruction code or as part of anapplication program or software product, or a combination thereof, whichis executed via the operating system. In one implementation, thetechniques described herein are implemented as computer-readable programcode tangibly embodied in one or more non-transitory computer-readablemedia 105. In particular, the present techniques may be implemented by aguidance generator 106 and a visualization unit 107. One or morenon-transitory computer-readable media 105 may include random accessmemory (RAM), read-only memory (ROM), magnetic floppy disk, flashmemory, and other types of memories, or a combination thereof. Thecomputer-readable program code is executed by processor device 104 toprocess data, images or image data acquired by, for example, imagingdevice 102, endoscopic instrument 130 and EM tracking system 135. Assuch, the computer system 101 is a general-purpose computer system thatbecomes a specific purpose computer system when executing thecomputer-readable program code. The computer-readable program code isnot intended to be limited to any particular programming language andimplementation thereof. It will be appreciated that a variety ofprogramming languages and coding thereof may be used to implement theteachings of the disclosure contained herein.

The same or different computer-readable media 105 may be used forstoring image datasets, a knowledge base and so forth. Such data mayalso be stored in external storage or other memories. The externalstorage may be implemented using a database management system (DBMS)managed by the processor device 104 and residing on a memory, such as ahard disk, RANI, or removable media. The external storage may beimplemented on one or more additional computer systems. For example, theexternal storage may include a data warehouse system residing on aseparate computer system, a picture archiving and communication system(PACS), or any other now known or later developed hospital, medicalinstitution, medical office, testing facility, pharmacy or other medicalpatient record storage system.

The imaging device 102 may be a radiology scanner, such as an C-armfluoroscopic X-ray or CT scanner, for acquiring image data. Theworkstation 103 may include a computer and appropriate peripherals, suchas a keyboard and display device, and can be operated in conjunctionwith the entire system 100. For example, the workstation 103 maycommunicate with the imaging device 102 so that the image data collectedby the imaging device 102 can be rendered at the workstation 103 andviewed on a display device.

The workstation 103 may communicate directly with the computer system101 to display processed image data and/or output image processingresults. The workstation 103 may include a graphical user interface toreceive user input via an input device (e.g., keyboard, mouse, touchscreen, voice or video recognition interface, etc.) to manipulatevisualization and/or processing of the image data. For example, the usermay view the processed image data, and specify one or more viewadjustments or preferences (e.g., zooming, cropping, panning, rotating,changing contrast, changing color, changing view angle, changing viewdepth, changing rendering or reconstruction technique, etc.), navigateto a particular region of interest by specifying a “goto” location,navigate (e.g., stop, play, step through, etc.) image volumes, and soforth.

Endoscopic instrument 130 may be inserted into the patient's body toexamine the interior of a tubular network (e.g., respiratory tract orairway). One example of an endoscopic instrument 130 is a bronchoscope,which includes an elongated rigid or flexible tube having a light sourceand a video camera for providing one or more real-time video images tothe computer system 101 from the bronchoscope' s tip. The bronchoscopemay further include a working channel (or sheath) through which one ormore instruments (e.g., biopsy or therapeutic catheters) may beinserted. For example, a catheter may be inserted in the working channelto take specimens from inside the lungs for diagnosis or to administertreatment (e.g., radiofrequency, laser or microwave ablation).

An electromagnetic (EM) tracking system 135 may optionally be providedto localize the endoscopic instrument 130 through the use ofelectromagnetic technology. The EM tracking system 135 may include asteerable guide catheter that contains a position sensor at its distaltip. The steerable guide catheter may be advanced through the workingchannel of the endoscopic instrument 130. The EM tracking system 135 mayprovide the tracked three-dimensional (3D) position (e.g., x, y, zcoordinates) of its distal tip when moving in the EM field. The 3Dposition information may be provided to the computer system 101 inreal-time. EM tracking system 135 may also communicate with imagingdevice 102.

It is to be further understood that, because some of the constituentsystem components and method steps depicted in the accompanying figurescan be implemented in software, the actual connections between thesystems components (or the process steps) may differ depending upon themanner in which the present framework is programmed. Given the teachingsprovided herein, one of ordinary skill in the related art will be ableto contemplate these and similar implementations or configurations ofthe present framework.

FIG. 2a shows an exemplary method 200 of image-based guidance by acomputer system. It should be understood that the steps of the method200 may be performed in the order shown or a different order.Additional, different, or fewer steps may also be provided. Someoptional steps are delineated with broken lines. Further, the method 200may be implemented with the system 100 of FIG. 1, a different system, ora combination thereof

At 202, imaging device 102 acquires 3D image data of a region ofinterest in the patient. Imaging device 102 may be a scanner or C-armsystem with a single imaging plane or multiple imaging planes. Forexample, imaging device 102 may include a rotating CT gantry covering atleast one pair of X-ray source and X-ray detector. In otherimplementations, imaging device 102 is a rotating optical CT gantrycovering at least one pair of light source and optical detector. Othertypes of imaging device 102 may also be used.

The region of interest may be any region that is identified for study,such as at least a portion of a tubular network (e.g., lungs, airway,bronchial tree). The 3D image data is acquired before (i.e.pre-operatively) or during a medical procedure (i.e. intraoperatively)performed on the patient. The 3D image data may be generated byacquiring tomographic image slices in different directions and combiningthem into a 3D image volume. FIG. 3 shows various exemplary views 302a-d reconstructed from a 3D CT image volume of a lung. In someimplementations, the 3D image data is registered to (or aligned with)the patient. Such image-to-patient registration may be rigid (e.g.,landmark-based or centerline-based) or nonrigid (or deformable).

Returning to FIG. 2 a, at optional step 203, a connection is establishedbetween imaging device 102 and EM tracking system 135. As discussedpreviously, the EM tracking system 135 may be optionally provided as anadditional information source for generating guidance to navigate theendoscopic instrument 130. If the EM tracking system 135 is used,guidance generator 106 may load data into the EM tracking system 135 andestablish a connection between the imaging device 102 and the EMtracking system 135. The guidance generator 106 may load, for example,3D image datasets (e.g., CT scan data of the region of interest) intothe EM tracking system 135. The connection between the imaging device102 and the EM tracking system 135 may be accomplished through a networkconnection to allow transfer of image and/or position coordinate data.

At 204, guidance generator 106 segments the region of interest in the 3Dimage data. One method of segmentation uses a threshold (e.g., setting alower or upper limit) for the Hounsfield unit (HU) of the image data.Pixels with density values in the preset range are carried over to thesegmented region of interest. Other methods of segmentation, such asregion growing or connected component techniques, are also useful. Thecenterline of the segmented region of interest may also be detected andrepresented by a set of points. The segmented region of interestreflects the path that the endoscopic instrument 130 is restricted tofollow. Labels may be assigned to identify the branches of the tubularnetwork (e.g., bronchial tree).

At 206, the starting and target points are identified in the segmentedregion of interest. The starting point is where the tip of theendoscopic instrument 130 is inserted or the position where thenavigation starts, while the target point is where the clinician plansto navigate the tip of the endoscopic instrument 130. The starting andtarget points may be selected by the clinician via, for example, a userinterface at workstation 103. For example, the user interface may begenerated and presented to the clinician to display a view of the 3Dimage data with the segmented region of interest and enable theclinician to digitally mark the starting and target points within thesegmented region of interest.

Guidance generator 106 may further generate suggestions of optimal viewsof the segmented region of interest to reduce airway ambiguity duringsuch user selection. The optimal view may be determined by, for example,calculating the viewing angle that provides a view of the segmentedregion of interest that is orthogonal to the direction of the X-raybeam. Such optimal views may be displayed via the user interface atworkstation 103.

At 208, imaging device 102 acquires 2D intraoperative image data of theregion of interest. The 2D intraoperative image data may be acquired atone or more specific time points during the medical procedure. When anEM tracking system 135 is used, the 2D intraoperative image data may beacquired at specific time points across, for example, a breathing cycleto generate a sequence of 2D intraoperative images. When the medicalprocedure is started, an endoscopic instrument 130 may be inserted intoa tubular network of the patient. For example, a bronchoscope may beinserted through the patient's nose, mouth or a tracheostomy, andadvanced through the bronchial (or airway) tree. A catheter may beinserted into the working channel of the bronchoscope to, for example,administer treatment or perform a biopsy for diagnostic purposes.

The 2D intraoperative image data may include, for example, one or more2D fluoroscopic images. FIG. 4 shows an exemplary 2D fluoroscopic image402 of a patient's lungs. The 2D fluoroscopic image 402 shows abronchoscope 404 inserted into a bronchial network during an ablationprocedure. A catheter 406 is inserted into the working channel of thebronchoscope 404 and extends beyond the tip of the bronchoscope 404 toadminister ablation for treatment of lung cancer.

Returning to FIG. 2 a, at optional step 209, when an EM tracking system135 is used, the EM tracked 3D position (e.g., x, y, z coordinates) ofthe distal tip of the endoscopic instrument 130 is acquired with each 2Dintraoperative image at each time point and translated into thecoordinate system of the imaging system 102. In some implementations,information from the EM tracking system 135 in the form of, for example,3D CT scan image data and the position of the tip of the EM trackedendoscopic instrument in this image data, is sent to the imaging system102. Such information may be fused with, for example, a cone beamcomputed tomography (CBCT) image acquired at the beginning of themedical procedure. Alternatively, a 3D to 2D fusion of the CT image withthe fluoroscopy images may be performed around the time of datatransfer. The translated 3D position information may then becommunicated to the imaging device 102 and/or computer system 101.

At 210, guidance generator 106 detects an endoscopic instrument 130 inthe 2D intraoperative image data and determines the centerline of theendoscopic instrument 130. As discussed previously, the endoscopicinstrument 130 may be, for example, a bronchoscope with a catheter andoptionally an EM steerable guide catheter with a position sensorinserted into its working channel. The endoscopic instrument 130 may bedetected using a region detection algorithm or any other suitable imageanalysis algorithm. The centerline may be calculated and represented bya locus of points that form the center path defining the approximatecenterline of the endoscopic instrument 130 and any catheter extendingbeyond the tip of the endoscopic instrument 130.

At 212, guidance generator 106 performs backprojection of the detectedendoscopic instrument 130 and corresponding centerline to generate athree-dimensional (3D) backprojected volume. In some implementations,backprojection may be performed for each 2D image in a sequence ofintraoperative images captured across different time points to generatea set of 3D backprojected volumes. Angiographic systems may becalibrated to enable 3D reconstruction (e.g., single view 3Dreconstruction). FIG. 5 illustrates an exemplary backprojection of thedetected bronchoscope and its corresponding centerline. The centerline502 of a bronchus is shown. The edges and centerline 504 of the detectedbronchoscope are backprojected to form a 3D volume 506. Thebackprojection may be performed using known system geometry.

Referring back to FIG. 2 a, at 214, guidance generator 106 determinesthe device path of the endoscopic instrument 130 based at least in parton the 3D backprojected volume and the 3D image data. The device path isthe most likely or correct passage of the tip of the endoscopicinstrument 130 from the starting point to the target point (or mostdistal segment) in the region of interest.

FIG. 2b shows an exemplary method 214 a of determining the device path.It should be understood that the steps of the method 214 a may beperformed in the order shown or a different order. Additional,different, or fewer steps may also be provided. Further, the method 214a may be implemented with the system 100 of FIG. 1, a different system,or a combination thereof

At 252, guidance generator 106 receives the 3D image data and thebackprojected volume. As discussed previously, the 3D image data may beacquired pre-operatively or intraoperatively by imaging device 102. The3D image data may include the segmented region of interest and itscorresponding centerline, as well as the starting and target points ofthe endoscopic instrument 130. The backprojected volume may be derivedfrom the 2D intraoperative image data of the region of interest and mayinclude the 3D backprojected endoscopic instrument 130 and itscorresponding centerline.

At 254, guidance generator 106 determines distances from the centerlineof each segment S of the segmented region of interest to the centerlineof the endoscopic instrument 130 in the backprojected volume. Moreparticularly, the shortest distance from each point along the centerlineof the segmented region of interest to the backprojected centerline ofthe endoscopic instrument 130 is determined, resulting in N data pointsor distances for each segment S. The segment S may be, for example, asegment of a bronchial network or any other tubular network.

At 260, guidance generator 106 determines all possible segment groupings(SG) from the set of segments S. Each SG is a set of one or moresegments that directionally connect the main segment of the tubularnetwork (i.e., where the starting point resides) to the determinedsegment that is most distal to the starting point (i.e., where thetarget point resides). All possible segment groupings satisfyingdirectional continuity may be considered. FIG. 6 shows exemplarycandidate segments 604 a-b. Candidate segments 604 a-b connect to themain bronchus 602 of the bronchial network.

At 262, guidance generator 106 determines the penalty function value foreach segment grouping SG to the backprojected centerline of theendoscopic instrument based at least in part on the N shortestdistances. The penalty function value may be calculated by determining anormalized sum of the shortest distances (P_(i)) of points along acenterline in a given segment grouping SG to the backprojectedcenterline of the endoscopic instrument and the deformation to beapplied to the segment grouping SG to contain the endoscopic instrument.The combined weighted sum of these calculations is called a penaltyfunction. An exemplary penalty function F is provided as follows:

F=w ₁(Sum(P _(i))/Length(SG))+w ₂Mag(D)   (1)

wherein D is the deformation field needed to create an intersectionbetween the segmented region of interest and the plane created bybackprojecting the endoscopic instrument; Mag(D) is the magnitude of thedeformation field D; w₁ and w₂ are predetermined weights; Sum(P_(i)) isthe sum of shortest distances (P_(i)) of all points i along a centerlineof a given segment SG to the backprojected centerline of the endoscopicinstrument, wherein i=1 to N; and Length(SG) is the length of the givensegment grouping SG. It should be noted that the larger the tubularsegment, the more difficult it is to deform and an additional penaltyweight may be introduced into the deformation field to represent this.

At 264, guidance generator 106 selects and returns the segment groupingwith the smallest penalty function value. This segment grouping forms atleast a portion of the device path of the endoscopic instrument and maybe presented to the user in, for example, a visualization.

FIG. 2c shows another exemplary method of determining the device path.It should be understood that the steps of the method 214 b may beperformed in the order shown or a different order. Additional,different, or fewer steps may also be provided. Further, the method 214b may be implemented with the system 100 of FIG. 1, a different system,or a combination thereof

At 282, guidance generator 106 receives the 3D image data of the regionof interest, 2D intraoperative image data of the region of interest andcorresponding backprojected volumes and EM tracked 3D positions of theendoscopic instrument 130. As discussed previously, the 3D image datamay be acquired pre-operatively or intraoperatively by imaging device102. The 3D image data may include the segmented region of interest andits corresponding centerline, as well as the starting and target pointsof the endoscopic instrument 130.

The backprojected volumes may be derived from a sequence of 2Dintraoperative image data of the region of interest acquired atdifferent time points. For example, the sequence of 2D intraoperativeimage data may be fluoroscopic images acquired during a breathing cyclewith the endoscopic instrument 130 in the bronchus not moving. Eachbackprojected volume may include the 3D backprojected endoscopicinstrument 130 and its corresponding centerline. Additionally, EMtracked 3D positions (e.g., x, y, z coordinates) of the distal tip andsupport points along the flexible shaft of the endoscopic instrument 130may also be received. The EM tracked 3D positions may correspond to thesequence of intraoperative 2D image data acquired at different timepoints.

At 284, guidance generator 106 selects the intraoperative 2D image Efrom the sequence of intraoperative 2D image data that corresponds to abackprojected endoscopic instrument tip that has the smallest distanceto its corresponding EM tracked 3D position.

At 286, guidance generator 106 translates the EM tracked 3D positions ofthe endoscopic instrument tip into a curve representation C withsupporting points. The curve representation may be, for example, aB-spline (or basis spline). A B-spline function is a combination offlexible bands that passes through a number of supporting points andcreates smooth curves.

At 288, guidance generator 106 updates the curve representation C tominimize the difference between the curve representation C and thebackprojected plane of endoscopic instrument derived from the selectedintraoperative 2D image E. This may be performed by modifying thesupporting points of the curve representation C to satisfy an objectivefunction that takes into account the overall tortuosity of theendoscopic instrument, the translation of each supporting point as wellas the stiffness of the tubular segment (e.g., the larger an airwaydiameter, the stiffer it is). An exemplary objective function isprovided as follows:

argmin(w ₁(C∩E)+w ₂ T(C)+w ₃Mag(D))   (2)

wherein T() is a function that calculates the bending energy containedin C; D is the deformation field needed to create an intersectionbetween the segmented region of interest and the plane created bybackprojecting the endoscopic instrument; Mag(D) is the magnitude of thedeformation field D; and w₁, w₂ and w₃ are predetermined weights.Supporting points of the curve representation C may be updated bydefining directions for each supporting point that minimizes theobjective function, moving a small amount in those directions. The 3Dtubular network model may then be deformed to include the updated curverepresentation C.

At 290, guidance generator 106 outputs the updated curve representation.Such updated curve representation may be used to form at least a portionof the device path of the endoscopic instrument that is presented to theuser. The updated curve representation and induced changes in the 3Dtubular network model may further be transferred to the EM trackingsystem 135, so as to improve overall EM tracking accuracy.

The method 214 b may be repeated as necessary. Once the endoscopicinstrument 130 is moved out of the tubular network, this approach workssimilarly, but the constraint of the tubular network in the objectivefunction may be dropped.

Referring back to FIG. 2 a, at 216, visualization unit 107 generates avisualization of the device path of the endoscopic instrument 130. Thevisualization may highlight (e.g., in different color or shading) thedevice path on image data of the region of interest. FIG. 7 shows anexemplary visualization. The device path 702 is highlighted in the imagedata. The most distal point 704 of the device path is visualized as thedetermined tip of the endoscopic instrument in a 3D viewing mode withrespect to the desired target position.

While the present framework has been described in detail with referenceto exemplary embodiments, those skilled in the art will appreciate thatvarious modifications and substitutions can be made thereto withoutdeparting from the spirit and scope of the invention as set forth in theappended claims. For example, elements and/or features of differentexemplary embodiments may be combined with each other and/or substitutedfor each other within the scope of this disclosure and appended claims.

What is claimed is:
 1. A system for image-based guidance, comprising: anelectromagnetic (EM) tracking system that acquires trackedthree-dimensional (3D) positions of a distal tip of an endoscopicinstrument insertable into a tubular network; and a computer systemcommunicatively coupled to the EM tracking system, wherein the computersystem includes a non-transitory memory device for storing computerreadable program code, and a processor in communication with the memorydevice, the processor being operative with the computer readable programcode to perform operations including segmenting at least a portion ofthe tubular network in 3D image data, detecting the endoscopicinstrument in two-dimensional (2D) intraoperative image data of thetubular network and determining a centerline of the detected endoscopicinstrument, performing backprojection of the endoscopic instrument andthe centerline to generate a 3D backprojected volume, determining adevice path of the endoscopic instrument based at least in part on the3D backprojected volume, the 3D image data and the tracked 3D positions,and generating a visualization of the device path.
 2. The system ofclaim 1 wherein the endoscopic instrument comprises a bronchoscope. 3.The system of claim 1 wherein the 2D intraoperative image data comprisesfluoroscopic images.
 4. The system of claim 1 wherein the EM trackingsystem acquires the tracked 3D positions corresponding to a sequence ofthe intraoperative 2D image data at different time points.
 5. The systemof claim 4 wherein the processor is operative with the computer readableprogram code to perform further operations comprising: selecting animage from the sequence of intraoperative 2D image data that correspondsto a backprojected endoscopic instrument tip with a smallest distance toa corresponding tracked 3D position; translating the tracked 3Dpositions into a curve representation; and updating the curverepresentation to minimize a difference between the curve representationand a backprojected plane of the endoscopic instrument derived from theselected image, wherein the updated curve representation forms at leasta portion of the device path.
 6. The system of claim 5 wherein the curverepresentation comprises a basis spline.
 7. An image-based guidancemethod, comprising: segmenting a region of interest in three-dimensional(3D) image data; detecting an endoscopic instrument in two-dimensional(2D) intraoperative image data of the region of interest and determininga centerline of the detected endoscopic instrument; performingbackprojection of the endoscopic instrument and the centerline togenerate a 3D backprojected volume; determining a device path of theendoscopic instrument based at least in part on the 3D backprojectedvolume and the 3D image data; and generating a visualization of thedevice path from a starting point to a target point.
 8. The method ofclaim 7 further comprises acquiring the 3D image data of the region ofinterest, wherein the region of interest comprises at least a portion ofa bronchial tree.
 9. The method of claim 7 further comprises acquiringthe 2D intraoperative image data of the region of interest during amedical procedure.
 10. The method of claim 9 wherein acquiring the 2Dintraoperative image data comprises acquiring one or more 2Dfluoroscopic images.
 11. The method of claim 7 further comprisesdetermining a centerline of the segmented region of interest.
 12. Themethod of claim 7 further comprises identifying, via a user interface,the starting point and the target point of the endoscopic instrument inthe segmented region of interest.
 13. The method of claim 7 whereindetermining the device path of the endoscopic instrument based at leastin part on the 3D backprojected volume and the 3D image data comprises:determining one or more segment groupings, wherein each segment groupingcomprises a set of one or more segments that directionally connect amain segment of the segmented region of interest to a most distalsegment of the segmented region of interest; determining one or morepenalty function values for the one or more segment groupings; andselecting at least one of the one or more segment groupings with asmallest penalty function value, wherein the selected segment groupingforms at least a portion of the device path.
 14. The method of claim 13further comprises determining shortest distances between a centerline ofa segment of the segmented region of interest and the centerline of theendoscopic instrument in the 3D backprojected volume and determining theone or more penalty function values based at least in part on theshortest distances.
 15. The method of claim 14 wherein determining theone or more penalty function values comprises calculating a weighted sumof the shortest distances of the one or more segment groupings and amagnitude of a deformation field to be applied to the one or moresegment groupings to contain the endoscopic instrument.
 16. The methodof claim 7 further comprises acquiring electromagnetic (EM) tracked 3Dpositions of a tip of the endoscopic instrument corresponding to asequence of the intraoperative 2D image data at different time points.17. The method of claim 16 wherein determining the device path of theendoscopic instrument based at least in part on the 3D backprojectedvolume and the 3D image data comprises: selecting an image from thesequence of the intraoperative 2D image data that corresponds to abackprojected endoscopic instrument tip with a smallest distance to acorresponding EM tracked 3D position; translating the EM tracked 3Dpositions into a curve representation; and updating the curverepresentation to minimize a difference between the curve representationand a backprojected plane of the endoscopic instrument derived from theselected image, wherein the updated curve representation forms at leasta portion of the device path.
 18. The method of claim 17 whereintranslating the EM tracked 3D positions into the curve representationcomprises translating the EM tracked 3D positions into a basis spline.19. The method of claim 17 wherein updating the curve representationcomprises modifying supporting points of the curve representation tosatisfy an objective function.
 20. One or more non-transitorycomputer-readable media embodying instructions executable by machine toperform operations, comprising: segmenting a region of interest inthree-dimensional (3D) image data; detecting an endoscopic instrument intwo-dimensional (2D) intraoperative image data of the region of interestand determining a centerline of the detected endoscopic instrument;performing backprojection of the endoscopic instrument and thecenterline to generate a 3D backprojected volume; determining a devicepath of the endoscopic instrument based at least in part on the 3Dbackprojected volume and the 3D image data; and generating avisualization of the device path.